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ABSTRACT 

This  Technical  Note  develops  an  application  of  the  theory  of  the 
Z-transform  to  the  problem  of  finding  the  nth  power  of  a  transition  matrix, 
motivated  by  the  need  of  such  a  solution  in  the  theory  of  Markov  Chains. 

To  illustrate  the  theory,  an  example  involving  vehicle  maintenance  is 
presented  in  detail.  The  results  of  the  example  may  be  compared  with  those 
of  Philip  M.  Morse,  "Markov  Processes",  in  Notes  On  Operations  Research, 
1959.  The  Technology  Press,  Massachusetts  Institute  of  Technology. 


Qualified  requesters  may  obtain  copies  of  this  Technical  Note  from  DDC 


INTRODUCTION 


Often  in  Markov  chain  problems  it  is  desired  to  find  the  nth  power,  P  , 
of  a  transition  matrix  P.  P  can  be  found  by  using  Z-transforms  as  follows: 

Let  II  be  the  row  vector, 
n 


n  *  f  n  ,  n  ,  . . .  ,  n 
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where  H.  is  the  probability  of  being  In  state  j  after  n  transitions.  (In 
genera  l-J  nbrackets  [  ]  will  enclose  a  matrix  or  vector.) 

Let  P  =  l  P. .  jbe  the  matrix  of  transit  ion  probab i 1 i t ies  of  going  from 
state  i  to  state'J  in  one  transition. 

Then  the  Markov  equation  is 

n  A  i  =  n  P  • 
n  +  I  n 

In  particular, 
ni  “  no  P  " 
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in  general , 

n  =  n_  pn  . 

n  0 

Let  the  Z-transform  be  defined  as 
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*{*„}  =  I  V"  ,  assuming  that  the  series  converges,  where 
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^  y  j’  is  a  sequence  of  numbers  y^,  y  ^  ,  ...  .  Since  Z  is  a  linear 

operator , 
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Multiply  by  z  «?nd  transposing  : 

Z  (nn)  -2Z(r‘n)  p  *  n0 

so  z  (  nn  )  •  [  I  .  2  p  ]  =  n0 

and  Z  ^  ^  ^  [I  -  2  P  ]  '  if  the  inverse  exists. 

Taking  inverse  transforms: 


so 
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Hence,  we  have  an  algorithm  for  computing  {  P"  }  ,  where  P 
sequence  of  matrices 

I  pij 

where  p..^  is  the  element  in  the  i th  row  and  jth  column  of  [  P  ]' 
2 . J  Hereafter,  the  generic  term  P  will  be  used. 


is  the 


k  =  1, 
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A  MAINTENANCE  EXAMPLE* 


A  bus  makes  N  runs  per  week.  Suppose  there  are  two  stages  in  the 
depreciation  of  the  bus;  in  stage  A  it  can  still  be  operated,  but  it  is 
apparent  that  it  will  soon  need  to  be  repaired;  in  stage  B,  it  cannot  do 
its  job  and  must  be  repaired.  'f  the  bus  is  repaired  when  it  is  discovered 
to  be  in  stage  A  of  depreciation,  the  repair  is  not  difficult,  but  if  it  is 
allowed  to  run  until  it  breaks  down  (stage  B)  the  repair  will  take  longer. 
The  decision  to  be  made  is  when  to  repair  the  bus. 

Suppose  the  deterioration  is  random,  such  that  if  it  is  in  good  con¬ 
dition  at  the  beginning  of  a  run  then  the  probability  it  will  still  be  in 
stage  A  at  the  end  of  the  run  will  be  a.  Similarly,  if  the  bus  is  in  stage 
A  at  the  beginning  of  a  run,  the  chance  that  it  will  reach  stage  B  by  the 
end  of  the  run  is  b.  For  simplicity,  let  the  length  of  time  to  repair  the 
bus  in  stage  A  be  the  same  length  of  time  it  takes  to  complete  one  run.  If 
stage  B  is  reached,  the  bus  will  be  out  of  commission  until  the  beginning 
of  the  following  week. 

Two  a  I ternat i ve  maintenance  doctrines  suggest  themselves:  Doctrine 
I  -  repair  if  the  bus  reaches  stage  A;  doctrine  2  -  wait  until  stage  B  is 
reached  to  make  repairs.  The  Markov  matrices  are: 

Doctrine  I  Doctrine  2 
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To  compare  the  two  doctrines,  two  questions  may  be  posed. 

(1)  What  is  the  expected  number  of  runs,  n  ,  the  bus  will  make  if  it  always 
starts  in  good  condition? 


This  is 


n 

z 


n  -  I  M 

r  V  n 

p.  over  all  states  k  in  which  the  bus  may 

n  =  0  k  =  1 


^Adapted  from  Morse,  Philip  M. ,  "Markov  Processes",  in  Notes  on  Operations 
Research,  1959.  The  Technology  Press.  Morse  uses  eigenvectors  to  obtain  his 
resu Its. 


3 


still  operate,  and  M  is  the  total  number  of  states. 


(2)  After  many  runs,  what  Is  the  probability  of  the  bus  not  breaking  down, 
i.e.,  what  is  the  utility  of  the  bus?  This  will  be  obtained  in  the  solution. 

SOLUT  ION 

Doctrine! . 


This  points  out  an  error  in  the  solution  of  this  example  in  Morse's 
article.  The  top  line  of  page  100  should  read: 
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Then  the  results  of  both  authors  agree: 


Now  n  can  be 
s 


computed: 
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Pn  can  be  represented  as  the  sum  of  two  matrices  as  in  equation  (1). 
The  effect  of  the  second  matrix  diminishes  as  n  gets  »0,  so  after  many 
runs  P  becomes  independent  of  n. 


1  +  a  1 


is  called  the  steady  state  transition  matrix. 


Note  that  each  row  is  identical.  The  elements  p..  of  any  row  i  represent 
the  steady  state  probabilities  of  being  in  state' j.  For  doctrine  1,  then, 
the  utility  of  the  bus  is  1/  1+a. 


Doctrine  2. 
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where  K  =  (1-2  ( 1 -a ) )  (1-2  (l-b))  ( I -2 ) . 
Using  partial  fractions: 
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Breaking  P  into  two  matrices: 
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0-a)n  ,  a 


(l-b)n  -  0-a)n 


0 

0 

0 


0 

0 

0 


b  (l-a)n  -a  (l-b)n 
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so,  if  N  is  large,  the  utility  of  the  bus  is  zero. 

RECOMMENDATIONS 

As  before  stated,  this  Technical  Note  is  not  a  development  of  the 
theory  of  Z-transforms ,  but  an  application  of  them  to  Markov  Chain  theory. 

Markov  Chains  are  useful  for  analyzing  any  probabilistic  or  random 
process  which  may  take  on  a  finite  number  of  possible  states.  Examples 
of  such  processes  are: 

1.  Machine  maintenance.  Here,  the  breakdown  of  operating  machines 
is  random.  The  states  may  be  "opera t  i ng1'  and  "non-operat i ng“. 

2.  Inventory  control.  Demands  for  maintenance  materials,  construc¬ 
tion  materials,  etc.  are  random.  The  states  of  an  inventory  system  might 
be  inventory  levels. 

3.  Replacement  of  items  that  fail.  Light  bulbs  in  a  building  fail 
at  random  times.  A  Markov  Chain  can  be  used  to  represent  this  process. 

The  states  of  the  process  are  perhaps  the  number  of  light  bulbs  which  have 
fa i led. 

Since  so  many  such  random  processes  occur  it  is  not  difficult  to  find 
other  applications  within  the  sphere  of  Shore  Operation  maintenance. 
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